Can the Search for “Fairness” Be Taken Too Far?

The many works devoted to the issue of fairness in language testing (e.g., Kunnan, 1999, 2000; Shohamy, 2001; Spolsky, 1981) testify to the field’s recognition of the importance of this issue. Brown (1996) defines fairness as “the degree to which a test treats every student the same or the degree to which it is impartial” (p. 31). The goal of language tests is to impartially measure individual test-takers’ language ability. If, however, performance on those tests is influenced by factors other than language ability, then bias is introduced into the measurement. Obviously, for ethical reasons, it is important to create fair and unbiased tests. But from a language testing standpoint, fairness and bias are also validity issues. Bachman and Palmer (1996) define construct validity as “the extent to which we can interpret a given test score as an indicator of the ability(ies), or construct(s), we want to measure” (p. 21). Messick (1989, 1996) describes how the introduction of construct irrelevant variance in a test threatens the construct validity of a test. Language tests are designed to measure an individual’s language ability, but if the test-taker’s employment background (for example) influences his or her performance on the test, then it is generally considered that construct irrelevant variance has been introduced into the measurement of the individual’s language ability. For example, a reading test is given to a group of test-takers from diverse backgrounds. The text describes how to pilot a helicopter. It would seem that test-takers who happen to be helicopter pilots would have an unfair advantage on this part of the test, and the other test-takers would be unfairly disadvantaged. The inferences made about a person’s reading ability based on the results of the test are of questionable validity. The bias in the test threatens the validity of the inferences made based on that test. A biased test cannot be a valid test. Subsequently, a test that is not valid can never be a fair test. Bias, viewed as measurement error, introduces construct irrelevant variance. Test developers seek to eliminate this construct irrelevant variance by minimizing test bias. Bachman (1990) lists some of the test-taker characteristics of an individual that might lead to biased assessments, including cultural background, background knowledge, cognitive characteristics, native language, ethnicity, sex, and age. Bachman and Palmer (1996), in describing language test development procedures, created a framework allowing test developers to systematically review their tests to investigate (and minimize) sources of bias in their tests. Large scale test developers have bias committees that examine test items and materials for potential sources of bias. However, in their zeal to eliminate bias from language tests, test developers might be going too far. As mentioned earlier, Messick (1989, 1996) describes how the introduction of construct irrelevant variance in a test (in this case, bias) can threaten the construct validity of that test. But Messick also lists another threat to the construct validity of a test: construct underrepresentation. In order to make valid inferences about the test-taker’s ability based on the test results, the items on a test must be adequately representative of the ability and content domain that is being assessed. If the test is too restrictive, and is not representative of the ability and content domain the test purports to assess, the validity of that measure is suspect. For example, if a test developer created a test that purported to assess a student’s overall proficiency in American history, but the test only had items that assessed American history since 1970, the